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EXECUTIVE SUMMARY 



In April 2003, Colorado passed H.B. 1 160, thus becoming the first state to enact a school 
voucher law in the wake of the landmark ruling in favor of school vouchers by the U.S. 
Supreme Court in June 2002. As expected, the Colorado law was immediately challenged 
in court. In December 2003, a state court declared the voucher program unconstitutional 
and, shortly thereafter, halted its implementation. While appeals have been filed, the legal 
status of the law remains unresolved as of this writing. Nonetheless, there are certain 
aspects about the creation, implementation and evaluation of this fledgling effort worth 
sharing with policymakers in other states that consider voucher laws. 



H.B. 1160 creates the Colorado Opportunity Contract Pilot Program. This program focuses 
on students from disadvantaged backgrounds who are enrolled in a low-performing school 
(in one of the state’s 1 1 lowest- performing school districts) and who are themselves per- 
forming poorly on state tests. To participate in the program, an eligible student must first 
apply to his or her school district. After being accepted, he or she must then apply to a non- 
public school that has chosen to participate in the program. 



Once a student is chosen to participate in the program and is accepted by a nonpublic 
school, the student’s parents must enter into a contract with the school district covering 
the terms and procedures of payment made by the district to the parents. Students also 
must take the statewide assessments each year, at the nonpublic school’s expense. H.B. 

1160 places a participation cap on the voucher program, with no more than 6% of a 
school district’s student enrollment for the previous school year allowed to participate at 
the height of the program. H.B. 1160 includes a sunset provision that repeals the law in 
2008, unless the legislature decides to renew it. 

State policymakers hope the program will improve the achievement of at-risk students 
who attend nonpublic schools, increase the satisfaction of at-risk students and their par- 
ents, stimulate public schools to improve the performance of remaining students, and 
increase support for public funding of schools. State policymakers also have several fears 
about the program, including that it will lead to a decrease in the achievement of at-risk 
students who attend nonpublic schools, fail to provide opportunities to targeted students, 
and have a negative impact on public schools as well as on society and democracy. Some 
of these attitudes are shared broadly, while others are not. 

According to several state and local leaders in Colorado, H.B. 1160’s delegation of authority 
for implementing and evaluating the voucher program is relatively weak. In some of the 
most interesting developments since the passage of the voucher law, such weaknesses have 
motivated state, local and private leaders to start three notable implementation and evalu- 
ation efforts. In one effort, the state board of education formed the Colorado Opportunity 
Contract Pilot Program Advisory Committee. This 16-member body is a diverse group of 
public, private and religious school representatives, some of whom support and some of 
whom oppose vouchers. Advisory committee members worked together to identify and 
solve practical implementation problems. 




In another effort, the Colorado Association of School Boards (CASB), instead of joining 
the legal challenge to the law, worked to help the 1 1 participating school districts address 
common problems involved in implementing it. CASB also worked in partnership with 
the above-mentioned advisory committee to develop various documents, including the 
applications that nonpublic schools are required to submit to school districts in order to 
participate in the program. 



Although H.B. 1160 contains some reporting requirements, many state, local and private 
leaders felt they were insufficient. In a particularly notable move, a coalition of voucher 
proponents and opponents - the Bighorn Institute for Public Policy, CASB and the 
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Colorado's 
voucher law 
was created 
as a pilot 
program, and 
this report 
takes that 
into account. 



Colorado Alliance for Reform in Education (CARE) - formed a partnership to create an 
evaluation design that will provide transparent data about the voucher program. 

As one of this partnership’s first activities, it invited a panel of the nation’s top voucher eval- 
uators to provide advice on how to evaluate the newly enacted voucher program to a group 
of state, local and private leaders in Colorado. As a result of this meeting, and acting on a 
suggestion by the researchers, the Bighorn Institute, CASB and CARE, along with Colorado 
Commissioner of Education William Moloney, agreed to move forward together in creat- 
ing the evaluation design. 

Whatever evaluations are produced through this and other efforts, they are sure to play 
some role in future debates over the state’s voucher program - assuming, of course, that it 
is eventually ruled constitutional. In the interviews conducted for this report, policymakers 
listed indicators that reflect all of the hopes and fears discussed above as data that will help 
inform future decisions. Several expressed skepticism, however, as to the ability or willing- 
ness of some policymakers - whether they support vouchers or oppose them - to change 
their position, regardless of the data such evaluations yield. 

The policymakers interviewed also emphasized the importance of test scores, including 
baseline information on student performance before they enter nonpublic schools. 
Determining whether to expand or even continue the voucher program, they said, should 
depend largely on evidence of improved test scores for participants, or neutral test scores 
combined with increased parent and student satisfaction. 

Surprisingly, several pro -voucher policymakers expressed opposition to expanding the pro- 
gram down the road, even with hard evidence of its success. Other voucher supporters can 
imagine supporting only a limited expansion, such as raising minimum income thresholds 
for participants or increasing the caps on the percentage of a district’s students that can 
participate in the voucher program while using the same eligibility standards. Still others 
described circumstances that might lead them to favor ending or scaling the program. 
These included major mismanagement and fraud, a degeneration of local politics in which 
choice was “fracturing communities” or a general lack of interest on the part of parents 
and students. 

Colorado’s voucher law was created as a pilot program, and this report takes that into 
account. To understand how the program will ultimately be judged to have succeeded or 
have failed, questions explored are about what the program should achieve, how that 
achievement should be measured and what mechanisms are in place to provide necessary 
information. 

This report begins by describing Colorado’s voucher program, and then takes a look at 
what state leaders on either side of the issue hope it will accomplish - or fear it will lead to. 
The report also examines the implementation and evaluation processes that are being put 
into place for the Colorado program, and concludes with a look at how evaluations might 
converge with voucher debates in the future. 

Strengthening the role of evaluations in future voucher programs will require crafting 
policies that call for: 

• Ensuring researchers have access to schools and students 

• Involving a variety of researchers reflecting different ideological and methodological viewpoints 

• Establishing an advisory board to provide continuity and long-term support for this research agenda. 

Doing so will improve the likelihood that policymakers ask important questions and that 
researchers provide credible answers, as well as generate a pool of information that all par- 
ties agree constitutes “what we know” about a given voucher program. 
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HE COLORADO VOUCHER PROGRAM 



In April 2003, Colorado passed H.B. 1 160, thus becoming the first state to enact a school 
voucher law in the wake of the landmark ruling in favor of school vouchers by the U.S. 
Supreme Court in June 2002. As expected, the law was immediately challenged in court. In 
December 2003, a state court declared the voucher program unconstitutional and, shortly 
thereafter, halted its implementation. While appeals have been filed, the legal status of the 
law remains unresolved as of this writing. Nonetheless, there are certain aspects about the 
creation, implementation and evaluation of this fledgling effort worth sharing with policy- 
makers in other states that consider voucher laws. 



H.B. 1160 creates the Colorado Opportunity Contract Pilot Program, which builds upon 
several school reform initiatives in the state over the past decade, including standards, 
assessments, accountability, open enrollment and charter schools. As one voucher supporter 
put it, the voucher law is one piece in the larger puzzle of school reform in Colorado, 
rather than a separate effort. 



H.B. 1160 applies to 1 1 school districts that had at least eight schools categorized by the 
state’s accountability system as “low” or “unsatisfactory” in the 2001-02 school year. It 
also allows a district to participate voluntarily in the program if its school board adopts a 
resolution stating its desire to do so. 

To receive a voucher in these 1 1 school districts, a student must be eligible to receive a free or 
reduced-cost lunch. There are also other requirements, depending on a student’s grade level: 

• Students entering or enrolled in kindergarten must lack overall learning 
readiness attributable to at least three significant family risk factors or their 
neighborhood school must be categorized as “low” or “unsatisfactory” in the 
state’s accountability system. 

• Students entering or enrolled in grades 1-3 must have been continuously 
enrolled in and attended a public school during the previous school year and 
must lack overall learning readiness attributable to at least three significant 
risk factors. Their neighborhood school also must be categorized as “low” or 
“unsatisfactory” in the state’s accountability system. 

• Students entering or enrolled in grades 4-12 must have been continuously 
enrolled in and attended a public school during the previous school year. They 
also must have performed at the “unsatisfactory” level in at least one academic 
area on the most recent statewide assessment, or in reading, writing or mathe- 
matics on the most recent college entrance exams. 

To participate in the program, an eligible student must first apply to his or her school 
district. Nonpublic schools that choose to participate in the program also must apply to 
a school district. After a student is accepted into the program, he or she must then apply 
to a participating nonpublic school. 




Once a student is chosen to participate in the program and is accepted by a nonpublic 
school, the student’s parents must enter into a contract with the school district covering 
the terms and procedures of payment made by the school district to the parents. Students 
also must take the statewide assessments each year, at the nonpublic school’s expense. 



School districts must pay parents the lesser of either the nonpublic schooTs per-pupil 
cost or: 



• 37.5% of the school district's per-pupil operating revenues (PPOR) if the eligible 
child is enrolled in kindergarten 

• 75% of the school district's PPOR if the eligible child is enrolled in grades 1-8 

• 85% of the school district's PPOR if the eligible child is enrolled in grades 9-12. 
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H.B. 1160 places a participation cap on the voucher program. For the 2004-05 school year, 
no more than 1% of a school district’s student enrollment for the previous school year may 
participate. The cap increases to 2% in 2005-06, 4% in 2006-07 and 6% for each school 
year thereafter. 

H.B. 1160 includes a sunset provision that repeals the law in 2008 - unless the legislature 
chooses to renew it. It requires the state auditor to produce a performance and financial 
audit of the program by January 1, 2008. It also requires participating school districts to 
provide a report to state policymakers by January 1, 2008. Finally, H.B. 1160 requires 
school districts to report the voucher program’s financial impact on them. 

With the passage of H.B. 1160, there are now six states that have enacted publicly funded 
voucher programs across the country - Colorado, Florida, Maine, Ohio, Vermont and 
Wisconsin. In addition, the U.S. Congress recently enacted a publicly funded voucher pro- 
gram for the District of Columbia. In reviewing the similarities and differences between 
Colorado’s program and the previously existing programs, several things stand out. 



Colorado's 
program. . . 
focuses on 
students from 
low-income 
families 
enrolled in 
low-performing 
schools. 



First, Colorado’s program targets students differently from the other voucher programs. 
Florida targets students in low-performing schools across the state as identified by the 
state’s accountability system. Ohio and Wisconsin target students from low-income families 
in one struggling school district (Cleveland and Milwaukee, respectively). And Maine and 
Vermont target students in communities without a public school, across the state. Like 
Florida’s voucher program, Colorado’s is part of the state’s accountability system. Colorado’s 
program, however, focuses on students from low-income families enrolled in low-performing 
schools - in one of the state’s 1 1 lowest-performing districts - or who are themselves per- 
forming poorly on state tests. 

Second, H.B. 1 160 is the first voucher law to be enacted since the federal government 
passed the No Child Left Behind (NCLB) Act in 2002. While NCLB requires school districts 
to provide students in low-performing schools with the opportunity to attend a higher- 
performing public school, it is apparent in Colorado - and elsewhere - that the potential 
population of students eligible for these opportunities in low-performing schools far 
exceeds the supply of seats in high-performing public schools. Although NCLB is silent 
on nonpublic-school choice, H.B. 1160 is the first attempt in light of NCLB to expand 
the supply of opportunities for these students to include seats in nonpublic schools. 

Third, a key component of accountability in any voucher program is expressed by parents 
"voting with their feet." Beyond this, though, there is scant accountability for student per- 
formance in most of the previously existing voucher programs. Perhaps in recognition of 
the increasing demands for accountability for student results in NCLB, Colorado's voucher 
program, like Florida's and Vermont’s, requires students to take the statewide assessments each 
year. In Colorado, though, these assessments will be taken at the nonpublic school's expense. 



Fourth, while many states claim to be “local control states,” there is state constitutional 
language in Colorado, as there is in Florida, that provides a legal basis for such a statement. 
According to Colorado’s constitution, local school boards “shall have control of instruction 
in the public schools of their respective districts.” This provision has been cited in several 
court cases over the years as a legal foundation for preserving the power of local school 
boards in several areas (including the recent state court decision that halted the voucher 
program). 

Given the legal context in Colorado, H.B. 1160 requires the direct involvement of school 
districts in ways that do not exist in voucher programs in other states. The law grants a 
considerable role to school districts in the student and nonpublic- school application 
processes. In addition, once a student is admitted into a nonpublic school, the law requires 
his or her parents to enter into a contract with the school district that covers the terms and 
procedures of payment made by the school district to the parents. 
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HOPES AND FEARS 



To understand how a pilot program and its evaluations should be designed to inform 
subsequent debates about the program, this section discusses the most significant hopes 
and fears that policymakers have for voucher programs, which encompass both direct and 
indirect effects of either a positive or harmful nature. This section also looks at the ways in 
which program evaluations may seek to measure the extent to which these hopes and fears 
are realized. 

HOPES 

Improve Student Achievement for At-risk Students Who Attend 
Nonpublic Schools 

The primary hope for the voucher program is that it will improve student achievement for 
at-risk students who attend nonpublic schools as part of the program. A related hope is it 
will decrease the achievement gap between higher- and lower-performing students. 

Based on the interviews with policymakers, it is apparent they hold a variety of definitions 
of the at-risk students that the program should target. While some see vouchers directly 
helping low- income children, others hope to target Colorado’s minority population. Most 
policymakers, though, describe the target population as children who are currently doing 
poorly in school. This last point is often couched in terms of students who are enrolled in 
schools where the performance is low, regardless of the individual voucher recipient’s pre- 
vious performance. In the aggregate, these student characteristics often overlap, which 
means that student performance is strongly correlated with family income, race and eth- 
nicity, and the concentration of low- income, minority and low-performing students in a 
particular school. 

The desired measures of student achievement for the voucher program are as multi-lay- 
ered as its target population. In the current climate of expanding testing and accountabili- 
ty, almost all policymakers interviewed believe performance on state assessments must be 
the primary measure of student achievement for voucher recipients. But a number of 
alternatives also are supported, including multiple measures of student achievement (e.g., 
grades or private schools’ standardized tests that are different than the state tests), meas- 
ures of life outcomes for older students (e.g., college attendance and college graduation) 
and measures of student engagement (e.g., attendance or participation in school activi- 
ties). These measures include raw gains by the targeted population as well as decreases in 
the performance gaps between targeted populations and other groups. 

Based upon the interviews, it seems that opponents look for a greater number of measures 
of a voucher program’s success, which leads some voucher supporters to question whether 
they are looking not just for information on how a program is implemented, but also 
hunting for ammunition for later political fights. Regardless of the intention, as one per- 
son opposed to vouchers explained, policymakers need “more than test scores, like how 
did the kids get there. . . how were they transported, if they were eligible for free and 
reduced-price lunch, did they eat lunch, and how did they eat it, and who paid for it, and 
who bought books, and paid other fees - a whole bunch more than simply state tests.” 

Increase the Satisfaction of At-risk Students and Families 

Some voucher advocates argue the only necessary measure of the voucher program’s direct 
impact is the satisfaction of the parents of children receiving vouchers. Other observers 
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“The more 
students they 
lose , the more 
money they 
lose. So they’ll 
be motivated 
to increase 
(performance).” 



believe that measures of student and family satisfaction are a necessary addition to student 
performance measures. Still others contend the best measure of a voucher program’s suc- 
cess will be the continued demand of families for vouchers. Thus initial rates of participa- 
tion, the rates of continued participation by families once they enter the program and the 
number of families that seek vouchers over and above the limits established by law are all 
posited as measures of program success. 

Regardless of whether measures of satisfaction are sufficient, one voucher supporter 
believes parental satisfaction is the best indicator of success: “The most important evaluator 
is the parent of the student. There can be positive outcomes for families and kids whether 
or not there is an increase in test scores.” These hopes for higher degrees of satisfaction tie 
into a philosophical belief that choice is a good worth pursuing for its own sake. Within 
this framework, the results of those choices for student performance are secondary to the 
benefit that comes from providing families with choice. 

Stimulate Public Schools To Improve Performance 
of Remaining Students 

The primary indirect hope for the voucher program is it will create competition between 
public and nonpublic schools, and thus stimulate public schools to enact reforms to 
improve the performance of the students that do not receive vouchers, or at least provide 
services and an environment that attracts parents back to public schools. In addition, some 
hope this increased “responsiveness” by public schools will discourage families from seek- 
ing vouchers in the first place. As one voucher supporter put it: “The more students they 
lose, the more money they lose. So they’ll be motivated to increase (performance).” 

To aid these school improvement efforts, a few voucher supporters hope the voucher program 
will increase the amount of money available per child in public schools. For example, if a 
district’s PPOR is $4,000, according to this line of reasoning, the value of the voucher for a 
4th grader - $3,000 (i.e., 75% of the district’s PPOR of $4,000) - is less than the costs of 
teaching the average child in most public schools. Thus if the students receiving vouchers 
are average students, their departure will leave the traditional public schools better off 
financially than if they had stayed - in this example, by $1,000 per student. A basic premise 
of this argument is if districts do not release or fire teachers as voucher recipients leave, the 
exit of voucher students will produce smaller class sizes for the remaining students. 

Discerning the direct and/or indirect impact of vouchers will be an extremely difficult 
enterprise. First, researchers will have to determine whether the voucher program pressures 
districts to improve. If districts actually “gain” money when they lose students through 
vouchers, and thus have no incentive to improve, the districts’ responsiveness will be 
undercut. Researchers will have to examine the extent to which districts and schools gain 
or lose money, as well as perceptions to any gain or loss on the part of leaders, administra- 
tors and teachers. 

If there is a sense among districts that vouchers provide an impetus for improvement, 
questions will emerge about whether the districts and their schools have the resources or 
capacity to improve after losing students to the voucher program. There also will be debate 
about how or whether aspects of public school administration or governance interfere with 
schools’ ability to make the changes they believe will facilitate their self-improvement. 
Finally, with the broad range of education reforms already underway (beyond vouchers), 
if public schools do improve, researchers and policymakers will still have a difficult time 
determining which reforms are responsible for the public schools’ improved performance. 
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One interviewee expressed the concern that basic data on the presence of pressures on dis- 
tricts to improve is not enough. The important question is whether districts respond to 
these pressures by performing better or worse. “[I need] to know the hard data,” he said. 
“Whether kids are learning, and not just learning, but they’re doing better than they would 
have in public schools and that the provision of services for children in the traditional 
public schools has not been negatively impacted.” 

Increase Public Support for Public Funding of Schools 

A final set of hopes for the voucher program involves the assertion that it will increase 
public support for public funding of schools. While this may appear paradoxical for 
those accustomed to arguments that vouchers will undermine public institutions, several 
observers, including those outside the voucher advocacy community, hope the provision 
of public funding for families to attend private schools will increase the population that 
has an interest in the public funding available to all schools - both public and nonpublic. 

According to one state-level official, “Not only will voucher recipients’ families want more 
money for public education, but parents of nonvoucher kids in private schools will also 
see the benefit of public dollars going to their children’s school, even if it comes through 
more voucher kids, and will become invested in public education.” 

FEARS 

Decrease Student Achievement for At-risk Students 
Who Attend Nonpublic Schools 

Fears about the Colorado voucher program include both direct and indirect harm that 
some believe it may generate. The primary fear is students receiving vouchers might not 
improve their achievement. This fear is exacerbated by anxieties that the private schools 
receiving voucher students will be lower in quality than the public schools that students 
leave, and that the value of the vouchers will not be enough to compensate private schools 
for the full costs of educating the new students. 

On a related note, some voucher proponents and opponents express concern that a dis- 
proportionate focus on parental satisfaction will limit information on other program 
measures such as student achievement. While acknowledging the importance of parental 
satisfaction, these individuals do not want to lose sight of the program’s effect on student 
performance. 

Fail To Provide Opportunities To Targeted Students 

Some individuals fear that students receiving vouchers, regardless of their performance, 
will not be the student population that policymakers had targeted to participate in the 
program. These concerns about whether voucher recipients will be the “right” students 
are shaped by fears of discrimination by private schools, as well as the inability, or unwill- 
ingness, of private schools to provide all the services that particular students may need 
to succeed. 

In particular, voucher opponents fear that students with disabilities or those who do not 
speak English will be denied access to private schools - or, if private schools enroll them, 
these students will not be provided with the level of service to which they are entitled in 
traditional public schools. Eventually, if students with special needs do not get the necessary 



The primary 
fear is that 
students 
receiving 
vouchers 
might not 
improve their 
achievement. 
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services, they may struggle and be encouraged to leave the private schools. Additional con- 
cerns over equitable access are based on broader fears of discrimination based on student 
characteristics like race, religion, income, performance or sexual orientation. 



“ Public 
education 
is this 

democracy’s 

greatest 

good.” 



Negatively Impact Public Schools 

Many voucher opponents argue public schools losing children to private schools will be 
unable to achieve any cost savings because of the difficulty of adjusting staffing patterns 
or otherwise reducing fixed costs. The fear is public schools will be so overwhelmed by 
such losses they will be irrevocably damaged, and unable to improve as a result. A related 
concern involves the extent to which existing governance arrangements and rules and regu- 
lations that constrain public institutions limit the ability of school districts to respond 
effectively to parental desires. With these obstacles still in place, the absence of resources 
could become more of a hindrance to success than a stimulus to improvement. 

Opponents also fear that the students receiving vouchers likely will be those with highly 
engaged parents, and that the students remaining in public schools will be those whose 
parents are less engaged in their education (or else they would have applied for vouchers 
themselves). This could leave the public schools with a student population that is more dif- 
ficult to teach than was the case before the voucher program. 

For some opponents, these concerns add up to fears about the long-term viability of public 
education. In their view, if public schools are unable to respond to vouchers by improving 
their performance, a cycle of disengagement and accelerating failure will emerge. The 
worse the public schools perform, the more the public will support further privatization, 
which in turn will lead to even fewer resources. 

Other fears are based not so much on the possible negative impacts of a targeted program 
as on the potential for expanding the program to a much larger, perhaps universal, scale. 
These fears are fueled by a suspicion that the initial program is a mechanism for beginning 
a process that will only end when a universal program is implemented. According to one 
voucher opponent, vouchers are “politically motivated to destroy the education system by 
showing that the public system is doing a bad job and people want vouchers. And this pilot 
program is designed to make vouchers palatable to people so they’ll accept a broader 
voucher program.” 

Negatively Impact Society and Democracy 

Some fear that vouchers will undermine the traditional concept of the public schools as 
protectors of a common heritage and a socialization process in which children of all back- 
grounds learn to work and live with other types of people - as opposed to a system in 
which parents choose schools where all the children are similar to their own and where 
diversity and community are limited. 

One person interviewed summed up this fear. In her view, “Public education is this democ- 
racy’s greatest good. The public aspect will be harmed in a way that will greatly detract 
from public education’s opportunity to serve all kids. ... “Education of children for and 
in a democracy - that is the public good that public education provides, and I’m not sure 
we can fulfill that if we have a fragmented system.” 
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IMPLEMENTATION AND 
EVALUATION EFFORTS 



According to several state and local leaders in Colorado, H.B. 1160’s delegation of authority 
for implementing and evaluating the voucher program is relatively weak. Such weaknesses 
have motivated state, local and private leaders to undertake three notable implementation 
and evaluation efforts. 

Colorado Opportunity Contract Pilot Program 
Advisory Committee 

For the most part, H.B. 1 160 is silent on who administers the voucher program. This was 
a somewhat deliberate decision by legislators to leave the state department of education 
out of the bill to avoid a fiscal note. This silence created a leadership vacuum that the state 
board of education decided to fill. As one person said, “H.B. 1160 didn’t say the state 
board can’t administer the program, so we unanimously decided to do it.” This decision 
partly grew out of the state board’s desire to become more active in the state’s school 
reform efforts. 

After gaining informal approval for this decision from the governor and key legislators, 
the state board formed the Colorado Opportunity Contract Pilot Program Advisory 
Committee. This 16-member body is a diverse group of public, private and religious 
school representatives, some of whom support and some of whom oppose vouchers. As 
one person interviewed pointed out, the state board was inclusive in forming this group, 
but they kept it small enough to get work done. 

Advisory committee members aired their opinions and concerns at the outset, and then 
worked together to identify and solve practical implementation problems. One of the first 
problems it tackled was the need to develop an application for nonpublic schools to sub- 
mit to school districts in order to participate in the voucher program. Other issues came 
up, such as what attachments to include with the application and how to handle building 
inspections for nonpublic school applicants. Several individuals interviewed said that the 
work of the advisory committee helped depoliticize implementation issues and built trust 
among people who typically don’t work together. 

Implementation Effort by the Colorado 
Association of School Boards 

The Colorado Association of School Boards (CASB) was opposed to H.B. 1160. Once it 
passed, however, CASB decided to help the 1 1 participating school districts implement the 
law, instead of joining the lawsuit against it. In an interesting turn of events, some voucher 
supporters provided financial assistance to this effort. As part of this deal, both CASB and 
these voucher supporters agreed to work together on a major evaluation effort (see 
“Evaluation Partnership” below). 

CASB’s work with the 1 1 participating school districts focused on addressing common 
problems in implementing the voucher program. It also worked in partnership with the 
Colorado Opportunity Contract Pilot Program Advisory Committee to develop various 
documents, including nonpublic school applications to school districts. 
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As one of this 
partnership’s 
first activities , 
it invited top 
voucher 
evaluators to 
provide state , 
local and 
private leaders 
advice on how 
to evaluate the 
new program. 



Evaluation Partnership (Bighorn Center for Public 
Policy, Colorado Association of School Boards, 
Colorado Alliance for Reform in Education) 

H.B. 1160 contains some reporting requirements. It requires the state auditor to produce a 
performance and financial audit of the program by January 1, 2008. It also requires partici- 
pating school districts to provide a report to state policymakers by January 1, 2008. The 
report must evaluate the academic performance of each eligible child who is enrolled in a 
nonpublic school, and include non-identifying individual student data on state tests and an 
analysis of individual student achievement, as well as similar data for those eligible chil- 
dren not selected in the lottery process to participate in the voucher program. Finally, H.B. 
1160 requires school districts to report the voucher program’s financial impact on them. 

While these efforts will be helpful to enhancing people’s understanding of the impacts of 
the voucher program, many state, local and private leaders felt they were insufficient. In a 
particularly notable move, a coalition of voucher proponents and opponent - the Bighorn 
Institute for Public Policy, CASB and the Colorado Alliance for Reform in Education 
(CARE) - formed a partnership to create an evaluation design that will provide transpar- 
ent data about the voucher program. 

As one of this partnership’s first activities, it invited top voucher evaluators to provide 
state, local and private leaders advice on how to evaluate the new program. The evaluators 
were Paul Peterson and William Howell of Harvard University, Kim Metcalf of Indiana 
University, John Witte of the University of Wisconsin-Madison and William Sanders of 
the SAS Institute Inc. 

The consensus among the researchers was that the “gold standard” in education research - 
the randomized field trial - is probably impossible to use in evaluating Colorado’s program 
because of certain aspects of the program’s design. They agreed that the “silver standard” in 
education research - a comparison study - is probably the best evaluation model in this 
case. 

The researchers also stressed the importance of providing evaluators with access to the pri- 
vate schools. For the evaluations of the Milwaukee, Wisconsin, and Cleveland, Ohio, 
voucher programs, such access was guaranteed in state policy. The researchers expressed 
concern about the impact of Colorado’s failure to do the same. The researchers also pro- 
vided advice on various issues and challenges involved in evaluating voucher programs in 
general, including suggested measures of program failure and success. 

As a result of this meeting, and at the suggestion of the researchers, the Bighorn Institute, 
CASB and CARE, along with the Colorado commissioner of education, volunteered to 
move forward together in creating an evaluation design for the voucher program. 

One positive aspect of the evaluation partnership is the bringing together of people with 
different viewpoints about the voucher program. In so doing, it has done a lot to take poli- 
tics out of the evaluation design. The members of the evaluation partnership are listening 
to and working with people on both sides of the issue, not just their side. It will be interest- 
ing to see how the coalition continues to work together to support the evaluation without 
influencing it. If the evaluation design establishes fair and measurable indicators, and com- 
pares apples to apples, it may prove to be a beneficial effort. 
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VOUCHER DEBATES AND 
EVALUATIONS: LOOKING AHEAD 



Whatever evaluations are produced through this and other efforts, they are sure to play 
some role in future debates over Colorado’s voucher program - assuming, of course, that 
it is eventually ruled constitutional. This section examines that issue, and presents some 
ideas for how policymakers and other leaders might strengthen the role of evaluations in 
future voucher programs. 



Policymakers’ Views on the Role of Evaluations 
in Future Voucher Debates in Colorado 



If a voucher program is enacted as a pilot program, as it was in Colorado, it stands to 
reason that eventually state leaders will examine the initial experience and decide whether 
to stop, continue or expand the program - based, ideally, on program evaluations. In the 
interviews, policymakers were asked to predict how these debates might play out in 
Colorado if the voucher program eventually is implemented. They were not asked to 
predict whether the program will succeed or fail. Instead, the information sought, regard- 
less of their initial position on the law, was what outcomes or data will lead them to vote 
to stop, continue or expand the program if they revisit it five years later. Their responses 
reflect many of the challenges discussed earlier about evaluating such a complex program. 

Policymakers listed indicators that reflect all of the hopes and fears discussed above as data 
that will help inform future decisions. Several of them voiced skepticism about the ability 
or willingness of some policymakers to change their positions on vouchers - regardless of 
what the evaluation data shows. One policymaker - who considered himself open-minded 
- said: “Most policymakers are biased. There aren’t that many policymakers willing to have 
[a substantive debate] .... There’s no room for the formation of good science in the policy- 
making process.” 

Policymakers emphasized the importance of test scores, including baseline information on 
the performance of students before they enter private schools. Many policymakers hope to 
base future decisions on performance by particular types of students. For example, one 
person suggested disaggregating results for the various “triggers” that make students eligi- 
ble, such as performing poorly on particular state tests or being enrolled in a low-perform- 
ing neighborhood school. 

Determining whether to expand or even continue the voucher program, policymakers 
said, should depend largely on evidence of improved test scores for participants or neutral 
test scores combined with increased parent and student satisfaction. But some believe 
higher test scores for children with vouchers must be viewed within the larger context of 
overall student performance. If a handful of voucher recipients have higher scores, but 
many more students without vouchers perform worse, several state officials believed that 
would provide a reason to limit or end the program. Policymakers opposed to vouchers 
also said future decisions should be based on the performance of, support for and 
resources available to public schools generally. 




Many pro -voucher policymakers believe high rates of participation and high levels of sat- 
isfaction for parents and students are enough reason to continue the program indefinitely, 
and that a level of demand that outpaces the availability of spots might justify expanding 
the program. 
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Surprisingly, several pro -voucher policymakers expressed opposition to expanding the pro- 
gram down the road, even with hard evidence of its success. Other voucher supporters can 
imagine supporting only a limited expansion, such as raising minimum income thresholds 
for participants or increasing the caps on the percentage of a district’s students that can 
participate in the voucher program while using the same eligibility standards. Still others 
described circumstances that might lead them to favor ending or scaling the program. 
These included major mismanagement and fraud, a degeneration of local politics in which 
choice was “fracturing communities” or a general lack of interest on the part of parents 
and students. 

Strengthening the Role of Evaluations 
in Future Voucher Programs 

To strengthen the role of evaluations in future voucher programs, policymakers should 
incorporate certain elements into the design of policies for these programs. These elements 
should ensure researchers have access to schools and students, those researchers have 
different ideological and methodological viewpoints, and an advisory board provides con- 
tinuity and long-term support for this research agenda. By putting in place these elements, 
state, local and private leaders will improve the likelihood that policymakers ask important 
questions and that researchers provide credible answers, as well as generate a pool of infor- 
mation all parties agree constitutes “what we know” about the voucher program. 

Guarantee Access: Researchers need access to private schools, to the students participating 
in a voucher program and to the students’ families. While the evaluators interviewed gave 
credit to Colorado for including a broad group of stakeholders in the early discussions 
regarding evaluations, they also were alarmed at the lack of authority for data- gathering in 
Colorado’s voucher law. They said researchers will need access to performance data from 
private schools, including the assessments private schools give to their students. In addition 
to test scores, researchers also will need the ability to survey voucher recipients and vouch- 
er applicants, as well as their families. They also will need to spend time in the private 
schools to see what they do and how they do it. 

If researchers must rely on the private schools’ voluntary participation in these research 
projects, the results will probably be suspect. Regardless of the results, opponents will dis- 
count them by alleging that only private schools that knew they were succeeding chose to 
give researchers data about, as well as access to, their schools. 

Guaranteeing access does not mean schools will necessarily be overwhelmed by researchers 
pursuing data. As one researcher noted, it is important to collect the right data, not all 
possible data. “You can collect bad data from millions of kids, and not say anything.” But 
researchers will need access, and it will be most effective if that access is granted early so 
that evaluations are designed with the best chances of answering key questions and estab- 
lishing key baseline data. 

Involve Experienced Researchers: One strategy to ensure the right data is gathered 
is to involve a variety of researchers in designing the evaluation from the outset of the 
voucher program. 
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As previously discussed, Colorado leaders brought together a broad set of researchers, 
stakeholders and policymakers to talk about program evaluation a few months after the 
voucher law was enacted. The scholars included people with broad experience researching 
vouchers, several of whom had initially reached different conclusions about the impacts of 
vouchers in their research. They also included evaluators with credibility outside the 
voucher debates who are respected for their methods of measuring student progress. 

While researchers may not be any more likely than policymakers to agree about all of the 
details of program evaluation, there are significant areas in which they will agree, as well as 
areas where their early involvement can strengthen later research. 

Form Evaluation Advisory Board: While researchers are used to long-term evaluations 
of complex programs, policymakers can often give intense attention to complex problems 
only for brief periods of time. Specific reforms or problems usually receive episodic atten- 
tion. As one policymaker explained, “We can’t babysit them after we pass them, and as 
much as I’m interested, I can’t monitor everything.” 

Obviously, there is a conflict between the timeframe researchers will need to do a good 
job of answering policymakers’ questions and the attention span of the policymakers who 
both frame the questions and whose support is necessary to guarantee backing for the 
ongoing research work. To maintain support for difficult research, and to keep such efforts 
focused on the right questions, state, local and private leaders should form an evaluation 
advisory board. This group should include senior and respected policymakers with a variety 
of positions on vouchers. Reputable scholars with appropriate experience should either 
serve on such a board or participate regularly in an advisory role. 

One state leader offered two recommendations. First, the board’s leaders should be senior 
politicians, preferably retired, to ensure board members do not use the board as a position 
from which to posture for later office. Another way to ensure open-mindedness is to include 
policymakers who previously voted in ways that were at odds with their party on vouchers. 

Second, an evaluation advisory board should include community leaders whose voices are 
valued in the state. These voices can include representatives of the business sector, as well 
as leaders from the Denver area’s minority communities. 

Colorado’s two efforts to convene diverse stakeholders around the voucher program 
were well- received. But it is unclear at this point whether either the implementation 
advisory committee or the evaluation partnership will have staying power, or whether 
they might wind up competing with each other. Still, both efforts are strong steps in the 
right direction. 
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CONCLUSION 




When the U.S. Supreme Court ruled in favor of the Cleveland voucher program, it cleared 
away a federal constitutional cloud that had hovered over voucher debates for a long time. 
Uncertainties remain, though, over whether vouchers will pass muster under individual 
state constitutions, as is currently occurring in Colorado. 

Over the next several years, as state budget crises recede and as policymakers respond to 
pressure from No Child Left Behind to expand choices for students in low-performing 
schools, the voucher debate will likely pick up steam. Notwithstanding the final legal ver- 
dict on Colorado’s voucher program, the program’s emphasis on low-income students in 
low-performing schools and districts, as well as the various implementation and evaluation 
efforts that have sprung up in response to it, may serve as a useful example for these 
emerging efforts. 
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